The present invention relates to a lane mark recognizing system for recognizing a lane mark on a traveling road, and to a picked-up image preprocessing apparatus incorporated in this lane mark recognizing system. Furthermore, the present invention relates to a vehicle traveling control system for controlling the vehicle according to the lane mark recognized by the lane mark recognizing system, and to a recording medium storing a program for executing the preprocessing of the picked-up image in a computer system.
To realize an automatic driving of a vehicle, it is necessary to accurately recognize a traveling road ahead of the vehicle. According to a conventionally developed recognizing method, a lane mark is recognized from a picked-up image of the traveling road ahead of the vehicle. The lane mark color is generally white or yellow which is discriminable from gray or black of the road color. For example, the unexamined Japanese patent publication No. Kokai 5-289743 discloses a method for recognizing the lane mark based on binary-coded data of an original picture. The unexamined Japanese patent publication No. Kokai 7-239996 discloses a method for recognizing the lane mark based on binary-coded data resultant from the edging processing applied to an original picture.
However, the recognizing method disclosed in the unexamined Japanese patent publication No. Kokai 5-289743 is based on a discrimination of a bright portion (i.e., lane mark) from a dark portion (i.e., road). All of brightness change is thus detected as indicating the lane mark. In other words, this conventional recognizing method erroneously recognizes the bright portions other than the lane mark. To eliminate this problem, an appropriate postprocessing is required after obtaining the binary-coded data for removing the noise data corresponding to the bright portions other than the lane mark.
Similarly, the recognizing method disclosed in the unexamined Japanese patent publication No. Kokai 7-239996 requires a complicated postprocessing. For example, the picked-up image may include another vehicles traveling ahead of the subjective vehicle. In such a case, the edging processed data comprise the edge data corresponding to the preceding vehicles as well as the lane mark. Thus, the postprocessing is required to discriminate the edge of the lane mark from the edges of the preceding vehicles.
On the other hand, the unexamined Japanese patent publication No. Kokai 3-194669 discloses a lane mark detecting filter capable of emphasizing a bright region surrounded by a dark region. The likelihood of the lane mark is judged based on the output of the lane mark detecting filter. The noise data is then removed according to the likelihood judgement.
According to the unexamined Japanese patent publication No. Kokai 3-194669, the brightness distribution is monitored along a lateral direction crossing the lane mark on the picked-up image. The lane mark is generally brighter than the road. Thus, when the above-described lane mark detecting filter is applied to the picked-up image, a positive filtering output is produced in response to the bright portion. Generation of the filtering output according to this conventional system is explained in more detail with reference to FIGS. 20A to 20D.
It is now assumed that the left half of an input image is a white region (brightness=200) and the right half is a gray region (brightness=100), as shown in FIG. 20A.
The filtering output is obtained by applying the lane mark detecting filter with respect to an objective pixel (X, Y). The lane mark detecting filter has a plurality of filtering parameters arranged in a matrix pattern including a total of four matrixes each consisting of 3xc3x973 pixels, with two matrixes (referred hereinafter to as xe2x80x9cnear-side matrixesxe2x80x9d) positioned next to both sides of the objective pixel and another two matrixes (referred hereinafter to as xe2x80x9cfar-side matrixesxe2x80x9d) positioned far from the both sides of the objective pixel. Each pixel has a positive pixel value (+1) in the two near-side matrixes. On the contrary, each pixel has a negative pixel value (xe2x88x921) in the far-side matrixes.
FIG. 20B shows a case where an objective pixel (Xa, Ya) resides in the white region so that all of the two near-side matrixes and the left far-side matrix are involved in the while region. Only the right far-side matrix is involved in the gray region. In this case, each of the near-side matrix produces 200*1*9=1,800. The left far-side matrix produces 200*(xe2x88x921)*9=xe2x88x921,800. The right far-side matrix produces 100*(xe2x88x921)*9=xe2x88x92900. Therefore, a summed-up filtering output becomes 1,800xc3x972xe2x88x921,800xe2x88x92900=900.
FIG. 20C shows another case where an objective pixel (Xb, Yb) resides in the gray region so that all of the two near-side matrixes and the right far-side matrix are involved in the gray region. Only the left far-side matrix is involved in the white region. In this case, each of the near-side matrix produces 100*1*9 =900. The right far-side matrix produces 100*(xe2x88x921)*9=xe2x88x92900. The left far-side matrix produces 200*(xe2x88x921)*9=xe2x88x921,800. Therefore, a summed-up filtering output becomes 900xc3x972xe2x88x92900xe2x88x921,800=xe2x88x92900.
FIG. 20D shows a filtered picture processed by the above-described lane mark detecting filter. As apparent from FIG. 20D, the lane mark detecting filter produces a positive filtering output in any transition from the bright region to the gray region. Thus, the positive filtering output may be produced in response to an erroneous bright portion other than the lane mark. For example, if a specific portion is as bright as the lane mark, a filtering output produced in a transition from this erroneous brighter portion to the road is quite similar to and not discriminable from the filtering output produced in the transition from the true lane mark to the road. Furthermore, if the brightness of this erroneous bright portion is fairly larger than that of the lane mark, the lane mark may be undesirably neglected due to the brightness difference between them.
In other words, the lane mark recognizing method of the unexamined Japanese patent publication No. Kokai 3-194669 is only reliable in an ideal condition where the lane mark has a clear contrast against the road with no erroneous bright portions. If there is any erroneous bright portion in the input image, the erroneous data must be removed in the detection of the lane mark.
An object of the present invention is to provide a picked-up image preprocessing apparatus capable of discriminating a bright portion corresponding to the true lane mark from other erroneous bright portions, performing preprocessing for obtaining only the data of the true lane mark, and improving the processing efficiency in the lane mark recognition.
Another object of the present invention is to provide a method for preprocessing the picked-up image used in the lane mark recognition.
Another object of the present invention is to provide a lane mark recognizing system incorporating the picked-up image preprocessing apparatus.
Another object of the present invention is to provide a vehicle traveling control system for controlling a vehicle based on the recognition result of the lane mark recognizing system.
Another object of the present invention is to provide a recording medium storing a program of a computer system for executing the above-described preprocessing.
The picked-up image preprocessing apparatus of the present invention is preferably applicable to a lane mark recognizing system for recognizing a lane mark on a traveling road ahead of the vehicle based on a picked-up image of the traveling road. For example, a CCD camera is installed on a vehicle body. The preprocessing apparatus of the present invention is applied to the image picked up by the CCD camera. The lane mark recognition is performed based on the preprocessed image.
The picked-up image preprocessing is performed in the following manner.
The picked-up image is scanned to generate an emphasized output responsive to a brightness change between a dark portion and a bright portion of the picked-up image, with a sign of the emphasized output reversing according to a transitional direction of the brightness change between the dark portion and the bright portion.
A picture resultant from the emphasized output contains a lane feature quantity obtained in a predetermined processing zone set on the picked-up image. The processing zone has a size applicable to a plurality of pixels of the picked-up image with a width wider than the lane mark. The lane feature quantity corresponds to a difference between an angular momentum of respective pixels about a central pixel in the predetermined processing zone and an accumulative absolute value of the respective pixels in the predetermined processing zone. The angular momentum is defined by vectorial quantities of pixel values of the respective pixels relative to the central pixel.
The lane mark is usually a while or yellow line drawn on a road which is recognized as a bright portion on the picked-up image. On the contrary, the lane mark may be a dark portion, such as a black lane mark printed on a white floor of a factory. For example, an automatic guided vehicle travels along a dark-color lane mark on a bright color floor of the factory.
In view of the presence of various kinds of lane marks, the picked-up image preprocessing is performed differently in each case.
In a first case where the lane mark is a white or comparable bright color lane mark, the lane mark is recognized as a bright portion in the picked-up image. In this case, a positive emphasized output is generated when the brightness change occurs in a transition from the dark portion to the bright portion, and a negative emphasized output is generated when the brightness change occurs in an opposed transition from the bright portion to the dark portion. The angular momentum of the respective pixels is positive when the angular momentum is detected in a clockwise direction relative to the central pixel in the predetermined processing zone.
In a second case where the lane mark is a black or comparable dark color lane mark, the lane mark is recognized as a dark portion in the picked-up image. In this case, a positive emphasized output is generated when the brightness change occurs in a transition from the bright portion to the dark portion, and a negative emphasized output is generated when the brightness change occurs in a transition from the dark portion to the bright portion. The angular momentum of the respective pixels is positive when the angular momentum is detected in a clockwise direction relative to the central pixel in the predetermined processing zone.
In a third case where the lane mark is a black or comparable dark color lane mark, the lane mark is recognized as a dark portion in the picked-up image. In this case, a negative emphasized output is generated when the brightness change occurs in a transition from the bright portion to the dark portion, and a positive emphasized output is generated when the brightness change occurs in a transition from the dark portion to the bright portion. The angular momentum of the respective pixels is positive when the angular momentum is detected in a counterclockwise direction relative to the central pixel in the predetermined processing zone.
In a fourth case where the lane mark is a white or comparable bright color lane mark, the lane mark is recognized as a bright portion in the picked-up image. In this case, a negative emphasized output is generated when the brightness change occurs in a transition from the dark portion to the bright portion, and a positive emphasized output is generated when the brightness change occurs in an opposed transition from the bright portion to the dark portion. The angular momentum of the respective pixels is positive when the angular momentum is detected in a counterclockwise direction relative to the central pixel in the predetermined processing zone.
According to the present invention, the lane feature quantity is used to appropriately define the likelihood of the lane mark. The lane mark is characteristic in that there is a band region having a predetermined width. When seen on the filtered picture, the lane mark has both ends whose pixel values (e.g., brightness) are substantially the same in absolute value but are opposite in sign (i.e., +or xe2x88x92). Accordingly, when the vectorial consideration is applied to the pixel values, a significant angular momentum is caused about a central pixel of a predetermined processing zone including the lane mark. More specifically, when the central pixel is located between the both ends of the lane mark in the width direction, the brightness change occurs at opposed sides of the central pixel. The positive pixel and the negative pixel are positioned at the opposed sides of the central pixel. Accordingly, the angular momentum of the same direction is caused about the central pixel. A large angular momentum is thus produced about the center of the lane mark.
On the other hand, an erroneous bright portion will cause a single brightness change when it does not have a width similar to that of the lane mark. If the brightness of the erroneous bright portion is identical with that of the lane mark, an angular momentum resultant from the erroneous bright portion will be half of that resultant from the lane mark. Thus, according to the present invention, the erroneous bright portion is easily discriminable from the erroneous bright portion.
However, if the brightness of the erroneous bright portion is two times the brightness of the lane mark, the angular momentum resultant from the erroneous bright portion will be identical with the angular momentum resultant from the lane mark. In this case, it is difficult to discriminate the lane mark from the erroneous bright portion.
To eliminate such a problem, the picked-up image preprocessing of the present invention further comprises a step of obtaining the accumulative absolute value in the predetermined processing zone.
According to the true lane mark, the accumulative absolute value becomes 0 because the positive and negative pixel values at the both sides of the lane marks are canceled.
On the other hand, according to the erroneous bright portion, no cancellation of the pixel values is expected when only one brightness change occurs. In other words, the accumulative absolute value resultant from the erroneous bright portion is substantially identical with the angular momentum (i.e., the pixel value itself).
The present invention obtains the lane feature quantity corresponding to the difference between the angular momentum of respective pixels about the central pixel in the predetermined processing zone and the accumulative absolute value of the respective pixels in the predetermined processing zone. The lane feature quantity for the true lane mark is substantially identical with the angular momentum which is a relatively large value. The lane feature quantity for the erroneous bright portion is substantially 0. Thus, the present invention makes it possible to surely discriminate the lane mark from the erroneous bright portion.
The above-described explanation is based on the bright lane mark. However, there will be a case where the lane mark is darker than the road. In this case, the sign (i.e., positive and negative) of the pixel values will be reversed. The angular momentum causes in the opposed direction. The resultant lane feature quantity for the true lane mark is a large negative value which is sufficiently discriminable from 0 of the erroneous dark portion. Thus, even in such a case, the present invention makes it possible to surely discriminate the lane mark from the erroneous dark portion.
In short, the present invention produces the filtered picture based on the lane feature quantity corresponding to the difference between the angular momentum of respective pixels about the central pixel in the predetermined processing zone and the accumulative absolute value of the respective pixels in the predetermined processing zone. The true lane mark is discriminable from the erroneous brightness change portion by comparing their lane feature quantities. As a simple processing, it is effective to convert each lane feature quantity into a binary value with reference to an appropriate threshold. Through this processing, all of noise data is removed and the data of the true lane mark can be surely obtained. This realizes an accurate recognition in the succeeding lane mark recognizing processing, and also improves the processing efficiency.
As apparent from the foregoing description, the inventors of the present invention introduce the xe2x80x9cangular momentumxe2x80x9d as an appropriate physical quantity representing the features of the lane mark, knowing that the xe2x80x9cangular momentumxe2x80x9d is seldom used in the field of the signal processing. In general, the xe2x80x9cangular momentumxe2x80x9d is the cross product of a vector from a specified reference point to a particle, with the particle""s linear momentum. The inventors of the present invention, however, carefully consider the fact that pixel values at the both ends of the lane mark image are opposite each other when observed in the brightness change appearing on the picked-up image. Such characteristic symmetrical disposition of the pixel values obtainable from the lane mark can be discriminated from a single brightness change inherent to the erroneous bright or dark portion. Thus, the inventors takes an approach to express the characteristics of the lane mark by the vectorial quantity quite similar to the angular momentum based on the positive and negative pixel values at both sides of the lane mark. In this respect, the inventors believe that the xe2x80x9cangular momentumxe2x80x9d is an appropriate physical quantity to express the features of the lane mark observed in the brightness change in the picked-up image.
The width of the lane mark is not always constant. A picked-up image may contain a plurality of lane marks different in width. A specific lane mark, for example drawn at a branch or merging portion, may be thicker than an ordinary lane mark. The width of the lane mark may be intentionally differentiated to discriminate one from another. For example, a plurality of paths are prepared on the floor of the factory using the automatic guided vehicle. In such a case, it is preferable to eliminate all of unnecessary lane marks.
A preferable embodiment of the present invention provides a dead zone with a predetermined width at the center of the predetermined processing zone so that the angular momentum and the accumulative absolute value are not obtained from this dead zone.
Providing this dead zone makes it possible to completely remove any lane mark narrower than the dead zone, because the brightness change occurring at both ends of the narrower lane mark cannot be sensed simultaneously by the processing zone with the center dead zone. In other words, the detectable lane marks can be substantially limited to the lane marks wider than the dead zone and narrower than the predetermined processing zone.
The camera of the lane mark recognizing apparatus is located at a predetermined altitudinal position of the vehicle which is generally higher than the road surface by a distance up to 2 meters. The camera can image-pick up a road view ahead of the vehicle according to the perspective representation. When a road having a constant width is image picked up by this camera, the road width on the picked-up image is wider in the lower region (i.e., at the near side) and narrower in the upper region (i.e., at the far side) due to the perspective representation.
Accordingly, the preferable embodiment of the present invention adjusts the predetermined processing zone so as to have a width widened at a lower side (i.e., near side) of the picked-up image and narrowed at an upper side (i.e., far side) of the picked-up image. With this adjustment, it becomes possible to change the width of the processing zone according to the perspective representation of the picked-up image. Similarly, it is preferable to adjust the dead zone so as to have a width widened at the lower side (i.e., near side) of the picked-up image and narrowed at the upper side (i.e., far side) of the picked-up image. With this adjustment, it becomes possible to change the width of the dead zone according to the perspective representation of the picked-up image.
More specifically, it is preferable that the width of the predetermined processing zone or the dead zone is determined based on a perspective transformation applied to the picked-up image.
In actual setting of the camera angle, the horizon is position at an altitudinal height spaced from the top by a predetermined vertical distance equivalent to ⅓ of the overall vertical length of the picked-up image. In this case, no road and no lane mark exists in the upper one-third region of the picked-up image. Therefore, it is preferable to set the width of the predetermined processing zone to zero at a predetermined upper region of the picked-up image where no lane mark is present. With this setting, it becomes possible to effectively eliminate unnecessary image processing.
According to the preferable embodiment of the present invention, the predetermined upper region including no lane mark is regarded as a region higher than an infinite position of the lane mark on the picked-up image.
According to the preferable embodiment of the present invention, to realize a simplified picked-up image processing, the predetermined processing zone extends in a horizontal direction of the picked-up image since the image processing is generally performed with reference to the horizontal direction and the vertical direction.
Especially, the lane mark on the picked-up image extends in the up-and-down direction although shown by the perspective representation. Thus, the horizontal processing zone crosses perpendicularly to the lane mark. This is preferable for effectively and accurately performing the picked-up image preprocessing to detect the lane mark.
However, the road does not always extend straight forward and flexibly changes its curvature according to the environmental conditions. If the vehicle is traveling on a curved road, the lane mark may extend in the horizontal direction rather than in the vertical direction on the picked-up image. In this respect, it is preferable that the predetermined processing zone is extendable in a vertical direction of the picked-up image as well as in the horizontal direction of the picked-up image.
In short, it is preferable that the predetermined processing zone is set along a direction normal to the lane mark on the picked-up image.
In this case, it may be possible to determine an optimum setting (e.g., optimum angle setting) of the predetermined processing zone based on each input picked-up image. However, this method requires a great amount of computations and takes a long time to process the picked-up image.
However, the picked-up image of the road and the lane mark does not cause a sudden and steep change. The present lane mark position is almost identical with or very close to the previously detected position. In view of the foregoing, to simplify the picked-up image preprocessing and reduce the substantial time and cost in this preprocessing, the preferable embodiment of the present invention sets the predetermined processing zone based on a previously detected lane mark position in the lane mark recognizing apparatus with which the picked-up image preprocessing apparatus is incorporated.
As described above, the present invention introduces the xe2x80x9cangular momentumxe2x80x9d as an appropriate physical quantity representing the features of the lane mark, knowing that the xe2x80x9cangular momentumxe2x80x9d is seldom used in the field of the signal processing. The inventors of the present invention, however, carefully consider the fact that pixel values at the both ends of the lane mark image are opposite each other when observed in the brightness change appearing on the picked-up image. Such characteristic symmetrical disposition of the pixel values obtainable from the lane mark can be discriminated from a single brightness change inherent to the erroneous bright or dark portion. Thus, the inventors takes the approach to express the characteristics of the lane mark by the vectorial quantity quite similar to the angular momentum based on the positive and negative pixel values at both sides of the lane mark. In this respect, the inventors believe that the xe2x80x9cangular momentumxe2x80x9d is an appropriate physical quantity to express the features of the lane mark observed in the brightness change in the picked-up image.
To express the above-described inventive approach in a different way, another aspect of the present invention provides a picked-up image preprocessing apparatus for applying predetermined preprocessing to a picked-up image of a traveling road ahead of a vehicle and installable in a lane mark recognizing apparatus for recognizing a lane mark on the traveling road based on the picked-up image. The picked-up image preprocessing apparatus comprises an image emphasizing means for scanning the picked-up image and generating an emphasized output responsive to a brightness change between a dark portion and a bright portion of the picked-up image, with a sign of the emphasized output reversing according to a transitional direction of the brightness change between the dark portion and the bright portion. An image output means is provided for producing a picture resultant from the emphasized output of the image emphasizing means. The picture contains a lane feature quantity obtained in a predetermined processing zone set on the picked-up image. The processing zone has a size applicable to a plurality of pixels of the picked-up image with a width wider than the lane mark. The image output means comprises a lane feature detecting means for detecting a symmetrical disposition of positive and negative pixel values of respective pixels about a central pixel in the predetermined processing zone, and an absolute value detecting means for obtaining an accumulative absolute value of the respective pixels in the predetermined processing zone. The image output means is for obtaining the lane feature quantity as an output difference between the lane feature detecting means and the absolute value detecting means.
Preferably, the above-described picked-up image preprocessing apparatus can be incorporated in a lane mark recognizing system so that the lane mark recognizing system can recognize the lane mark based on the lane feature quantity obtained by the image output means of the picked-up image preprocessing apparatus.
Furthermore, it is preferable to incorporate the picked-up image preprocessing apparatus and the lane mark recognizing apparatus into a vehicle traveling control system so that the vehicle traveling control system can control the traveling of the vehicle according to the lane mark recognized by the lane mark recognizing apparatus.
The present invention is not limited to the (white or yellow) lane mark on a general road and therefore can be applied to various guide lines printed on the floor of a factory using an automatic guided vehicle. The road defined in this invention includes this kinds of paths provided on the factory floor.
Furthermore, it is preferable to store the above-described preprocessing as a program of a computer system. The program can be stored in a portable or handy recording medium, such as a floppy disk, a MO (magneto-optical) disk, a CD-ROM, a DVD (i.e. digital versatile disk), and a hard disk. Moreover, the program can be stored in a ROM or a backup RAM which is incorporated beforehand in a computer system.
According to the preferable embodiment of the present invention, there is a first image filter having filtering parameters arranged in a matrix pattern applicable to a plurality of pixels on the picked-up image. The first image filter produces an emphasized output responsive to a brightness change between a dark portion and a bright portion of the picked-up image, with a sign of the emphasized output reversing according to a transitional direction of the brightness change between the dark portion and the bright portion. There is a second image filter having filtering parameters having a predetermined processing zone applicable to a plurality of pixels on the picked-up image. The second image filter produces an output representing an angular momentum of respective pixels about a central pixel of the predetermined processing zone. The angular momentum is defined by vectorial quantities of pixel values of the respective pixels relative to the central pixel. There is a third image filter having filtering parameters having a predetermined processing zone applicable to a plurality of pixels on the picked-up image. The third image filter produces an output representing an accumulative absolute value of respective pixels about a central pixel of the predetermined processing zone. The lane feature quantity is detected as a difference between the output of the second image filter and the output of the third image filter.
The first image filter and the second image filter can be replaced by a composite filter capable of producing the equivalent output. Similarly, the first image filter and the third image filter can be replaced by a composite filter having the equivalent output.
According to the preferable embodiment of the present invention, the preprocessing of the picked-up image is performed according to the following steps. In a first step, an emphasized output is generated in response to a brightness change between a dark portion and a bright portion of the picked-up image with a sign of the emphasized output reversing according to a transitional direction of the brightness change between the dark portion and the bright portion. In the next step, a predetermined processing zone is set on the picked-up image. The processing zone has a size applicable to a plurality of pixels of the picked-up image with a width wider than the lane mark. An angular momentum of respective pixels is obtained about a central pixel in the predetermined processing zone. The angular momentum is defined by vectorial quantities of pixel values of the respective pixels relative to the central pixel. An accumulative absolute value of the respective pixels is obtained in the predetermined processing zone. Then, in the next step, a lane feature quantity is obtained as a difference between the angular momentum and the accumulative absolute value of the respective pixels. A picture containing the lane feature quantity is thus produced.
For example, the first image filter is applied to the picked-up image for producing the emphasized output. The second image filter is applied to the emphasized output of the first image filter for producing the output representing the angular momentum of respective pixels. The third image filter is applied to the emphasized output of the first image filter for producing the output representing the accumulative absolute value of respective pixels.